On the Complexity of Learning with Kernels

A well-recognized limitation of kernel learning is the requirement to handle a kernelmatrix, whose size is quadratic in the number of training examples. Many methodshave been proposed to reduce this computational cost, mostly by using a subset of thekernel matrix entries, or some form of low-rank matrix approximation, or a randomprojection method. In this paper, we study lower bounds on the error attainable bysuch methods as a function of the number of entries observed in the kernel matrix or therank of an approximate kernel matrix. We show that there are kernel learning problemswhere no such method will lead to non-trivial computational savings. Our results alsoquantify how the problem difficulty depends on parameters such as the nature of theloss function, the regularization parameter, the norm of the desired predictor, and thekernel matrix rank. Our results also suggest cases where more efficient kernel learningmight be possible.